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Abstract 

Key features of the school environment can have a significant impact on teachers’ effective use of evidence-based practices 
(EBP), yet implementation-specific organizational constructs have rarely been studied in the education sector. This study 
examined three aspects of the organizational implementation context (implementation leadership, climate, and citizen- 
ship behavior), which have been conceptualized and validated in other service settings. Focus groups with central office 
administrators, principals, and teachers were conducted to understand the applicability and conceptual boundaries of these 
organizational constructs in schools. Focus group transcripts were coded, and the results indicated both similarities and 
differences in their conceptualizations of implementation leadership, climate, and citizenship behavior in school. The data 
indicated that: (1) implementation leadership was largely present in schools with the addition of Distributed Leadership; (2) 
two implementation climate constructs were most clearly present (i-e., Focus on EBP and Educational Support for EBP) and 
two additional constructs (i.e., Existing Support to Deliver EBP and Prioritization of EBP) emerged as part of this construct; 
and (3) implementation citizenship behavior (Helping Others and Keeping Informed) was consistently acknowledged across 
schools and two new components emerged (i.e., Information Sharing and Observation/Feedback). Recommendations to 
researchers and community stakeholders are discussed. 


Keywords Schools - Implementation - Evidence-based practices - Leadership - Climate - Citizenship behavior 


Introduction 


Schools are the most common setting for the delivery of 
mental health services to youth in the USA (Costello, He, 
Sampson, Kessler, & Merikangas, 2014; Farmer, Burns, 
Phillips, Angold, & Costello, 2003; Langer et al., 2015; 
Lyon, Ludwig, Vander Stoep, Gudmundsen, & McCauley, 
2013; Merikangas et al., 2011). There is immense pressure 
for public schools to adopt a continuum of evidence-based 
practices (EBP), defined as those with the best research 


Jill Locke 

jjlocke@uw.edu 

University of Washington, Seattle, USA 
University of Minnesota, Minneapolis, USA 
3 Univeristy of Central Florida, Orlando, USA 
University of California, San Diego, USA 


evidence, clinical expertise, and that cater to patient prefer- 
ence and culture (APA Presidential Task Force on Evidence- 
Based Practice, 2006; Cook & Odom, 2013), across univer- 
sal, targeted, and intensive levels of services to improve the 
mental health of youth (Fixsen, Blase, Metz, & Van Dyke, 
2013; Odom, Cox, Brock, & National Professional Develop- 
ment Center on ASD, 2013). However, successful adoption, 
delivery, and sustainment of EBP in schools are fraught with 
challenges (Owens et al., 2014). When EBP are adopted in 
schools, only 25-50% are implemented with fidelity (i.e., 
implemented as intended) (Cook & Odom, 2013; Gottfred- 
son & Gottfredson, 2001). Delivery of EBP with poor fidel- 
ity is unlikely to be effective in changing youth outcomes 
(Durlak & Dupre, 2008). This is a critical issue that results 
in significant wasted resources and weakens the potential of 
schools to promote youth’s mental health outcomes. 
Recent research has examined the role of organizational 
constructs in the successful use of EBP in other service 
sectors such as specialty mental health and child welfare 
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(e.g., Aarons et al., 2012; Beidas et al., 2013, 2014, 2016a; 
Bonham, Willging, Sommerfeld, & Aarons, 2014; Ehrhart, 
Aarons, & Farahnak, 2015; Powell et al., 2017), which also 
have relevance to schools (Forman et al., 2013; Hoagwood 
& Johnson, 2003; Langley, Nadeem, Kataoka, Stein, & Jay- 
cox, 2010; Owens et al., 2014). Organizational constructs 
may play a critical role in the implementation of school- 
based services that maximize school mental health. Lyon 
et al. (2018) used the Exploration, Preparation, Implemen- 
tation, Sustainment framework (EPIS; Aarons, Hurlburt, & 
Horwitz, 2011), a four-phase, prospective implementation 
framework that delineates the nested structure of outer (i.e., 
larger system-level) and inner (i.e., building level) contextual 
constructs that drive EBP implementation efforts, to define 
the organizational implementation context (OIC). The OIC 
represents malleable constructs specific to the inner context 
(i.e., microsystemic factors associated with a given school 
building) that influence successful EBP implementation in 
schools (Lyon et al., 2018). The OIC represents the imme- 
diate setting in which implementation takes place. As such, 
OIC constructs are most proximal to and likely to have an 
influence on implementer behavior than other factors outside 
that are farther removed from the place where implemen- 
tation happens (Jacobs, Weiner, & Bunger, 2014; Malloy 
et al., 2015). The OIC captures three core organizational 
constructs: implementation leadership, implementation 
climate, and implementation citizenship behavior. These 
OIC constructs have been less frequently studied in schools 
but are likely to serve as proximal contextual indicators of 
behavior change among school-based practitioners (e.g., 
teachers, school counselors, licensed school-based mental 
health provider) who are responsible for the delivery of 
different services within a multi-tiered continuum of care 
(Bruns et al., 2016). 

Implementation leadership refers to specific behaviors 
that leaders perform to support the implementation of EBP 
(Aarons, Ehrhart, & Farahnak, 2014a). Prior research has 
identified five dimensions of implementation leadership 
that combine to influence implementation outcomes in a 
given organizational context: organizational leaders’ being 
knowledgeable about the identified EBP, supportive behav- 
iors directed toward frontline providers, proactive anticipa- 
tion and problem solving regarding barriers that are likely 
to arise during the implementation process, perseverance 
with staying the course with implementation despite barri- 
ers that may arise, and availability to support and trouble- 
shoot issues with frontline providers (Aarons et al., 2014a; 
Ehrhart et al., 2018). Implementation leadership can operate 
as a critical construct of the OIC that establishes a specific 
climate in a school that is conducive to EBP adoption, deliv- 
ery, and sustainment across multiple tiers (Aarons, Ehrhart, 
Farahnak, & Sklar, 2014b; Aarons, Ehrhart, Torres, Finn, 
& Beidas, 2017). 
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Implementation climate is more specific than general 
school climate. While school climate reflects how indi- 
viduals within schools perceive and ultimately describe 
the environment in their school, implementation climate 
reflects staff’s shared perceptions of the policies, practices, 
and procedures supporting EBP implementation, as well 
as the kinds of behaviors that are expected, supported, and 
rewarded as part of the implementation process (Ehrhart, 
Aarons, & Farahnak, 2014). Ehrhart et al. (2014) identi- 
fied six dimensions of implementation climate including the 
organization’s: Focus on EBP, Educational Support for EBP, 
Recognition for EBP, Rewards for EBP, Employee Selection 
for EBP, and Selection for Openness. Organizations with 
low levels of implementation climate fail to demonstrate that 
EBP implementation is a valued endeavor as there is limited 
focus on EBP, support provided, and/or forms of recogni- 
tion and acknowledgment for staff who invest in EBP imple- 
mentation. Although the two constructs are distinct, imple- 
mentation climate is primarily driven by implementation 
leadership and is supported by specific leaders’ behaviors 
that communicate those norms and expectations and allocate 
necessary resources (e.g., protected time, materials, money) 
to demonstrate the organization’s values within a given con- 
text (Aarons et al., 2014a). Existing research suggests that 
such focused or strategic climates are more related to spe- 
cific outcomes rather than more molar or general measures 
(Schneider, Ehrhart, & Macey, 2013). Implementation cli- 
mate may be a critical construct to examine in schools to 
help support the use of EBP (Locke et al., 2016). 

Lastly, organizational citizenship behavior is exhibited 
when employees go “above and beyond” their core job 
aspects or standard “call of duty” to further the mission of 
the organization (Organ, Podsakoff, & MacKenzie, 2005). 
Applying this concept to the goal of EBP adoption and sus- 
tainment, implementation citizenship behaviors are those 
that demonstrate a commitment to EBP within the organi- 
zation whereby individuals strive to keep informed about the 
EBP and offer support to their colleagues who are attempt- 
ing to deliver the EBP with fidelity (Ehrhart et al., 2015). 
Ehrhart et al. (2015) posit two dimensions of implemen- 
tation citizenship including: Helping Others and Keeping 
Informed. Implementation citizenship behaviors serve as 
a hypothesized mediator by which implementation leader- 
ship and implementation climate exert their influence on 
implementation success (Ehrhart, Aarons, Torres, Finn, & 
Roesch, 2016). Implementation citizenship captures specific 
behavior changes in school staff (e.g., helping a colleague 
with EBP implementation) and may mediate the influence 
of implementation leadership and implementation climate 
on successful EBP use. 
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Implementation research in other service sectors is generally 
more advanced than in schools (Sanetti, Knight, Cochrane, 
& Minster, in preparation). The multidisciplinary field of 
implementation science creates opportunities for research- 
ers and practitioners to take advantage of existing findings 
and products from other service sectors, like health care, 
specialty mental health, and child welfare, by adapting and 
examining the application and generalizability of prior find- 
ings in novel contexts such as schools. For example, the 
ways in which the OIC constructs operate in the child wel- 
fare sector may look and operate differently in schools and 
there are few studies investigating implementation leader- 
ship, climate, and citizenship behavior within the context of 
EBP implementation in schools (Locke et al., 2016; Lyon 
et al., 2018). The absence of research in this area may be due 
to unknown relevance and appropriateness of OIC constructs 
to the education sector without systematic consideration of 
the school context (Lyon et al., 2018). It is possible that the 
manifestation of implementation leadership, climate, and 
citizenship in schools differs from the ways in which it is 
measured in other service contexts (e.g., hospitals, commu- 
nity mental health agencies). In addition, each of the OIC 
constructs may be perceived differently by individuals in dif- 
ferent school roles (central office administrators, principals, 
teachers). Understanding the perspectives of different stake- 
holders can provide a more nuanced understanding of OIC 
constructs (Beidas et al., 2016b) in schools and may point 
to the need for a shift in thinking with regard to EBP imple- 
mentation. Lastly, the boundaries of the OIC constructs may 
be more or less expansive in schools than in other settings 
in which they were initially conceptualized, resulting in the 
addition or deletion of specific subconstructs. This study 
may help guide future EBP implementation in schools. 


Purpose of this Study 


The current study occurred in the context of a larger spon- 
sored project to adapt a suite of measures capturing OIC 
constructs for use in schools to support data-driven continu- 
ous improvement of evidence-based universal supports tar- 
geting student social, emotional, and behavioral outcomes. 
The purpose of this study was to use qualitative methods 
to examine how each of the OIC constructs (implementa- 
tion leadership, implementation climate, and implementa- 
tion citizenship behavior) is conceptualized in schools to 
determine whether the OIC constructs have different mean- 
ing and application than other service sectors in which they 
have been assessed (e.g., community mental health and child 
welfare), as well as generate potential relevant subconstructs 
that fall under the three broader OIC constructs. Given 
that school-based mental health involves the delivery of a 
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continuum of services across universal, targeted, and inten- 
sive levels of care (Bruns et al., 2016), this study included 
different groups of educational stakeholders (i.e., district 
administrators, principals, teachers) involved in varying 
ways with implementation. For example, central administra- 
tors often manage and track implementation across schools, 
principals oversee and support implementation within their 
own schools, and teachers are the primary implementers of 
universal supports or collaborators on more intensive mental 
health interventions. The inclusion of different stakeholder 
groups also allowed us to examine whether perspectives of 
the OIC constructs varied as a function of group. 


Methods 
Setting and Participants 


This study occurred as part of a larger federally funded pro- 
ject examining the iterative adaptation and validation of spe- 
cific measures capturing key constructs of the school OIC. 
The university institutional review board and each partici- 
pating school district approved the study. All participants 
provided informed consent prior to their participation. We 
first contacted a central administrator from each of the two 
partnering school districts. Next, the central administrator 
provided the names of 6-8 staff for each of three stakeholder 
groups: central administrators, elementary school principals, 
and elementary school teachers. Our recruitment process 
resulted in a total of 37 individuals (16 central administra- 
tors, 10 elementary school principals, and 11 elementary 
school teachers) from two relatively large school districts in 
the Northwestern USA. The two school districts are socioec- 
onomically (approximately 26% of students in these districts 
qualify for free and reduced lunch) and racially/ethnically 
diverse (41.3% White; 26.8% Asian; 11.6% Hispanic/Latino; 
9.6% Multiethnic; 8.7% African American; 0.3% American 
Indian or Alaskan Native; 0.2% Native Hawaiian). Together, 
these two school districts have 91 elementary schools. The 
sample was predominantly female (n= 29, 78.38%) with an 
average age range of 35-44 years old. Age bands were col- 
lected rather than exact ages to accommodate our participant 
population who felt more comfortable reporting ranges. The 
ethnic backgrounds of participants were as follows: 75.68% 
white, 10.81% multiracial/multiethnic, 5.40% Asian, 2.70% 
African American, 2.70% American Indian/Alaskan Native, 
and 2.70% Native Hawaiian or Other Pacific Islander. Their 
highest educational attainment was as follows: 8.11% had a 
doctoral degree, 86.49% had a master’s degree, and 5.40% 
had a bachelor’s degree. See Table | for demographic 
information. 
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Table 1 Demographics Variable 


Age 

18-24 years old 
25-34 years old 
35-44 years old 
45-54 years old 
55-64 years old 
65-74 years old 

75 years or older 
Gender 

Male 

Female 
Race/ethnicity 

Am. Indian or Alaskan Native 
Asian 

Black or African Am. 


Native Hawaiian or Other Pacific Islander 


White or Caucasian 
Multiethnic 
Advanced degree 
Bachelor’s degree 
Master’s degree 
Doctoral degree 
Total 


Whole sample Central admin Principals Teachers 
1 (2.70%) 0 (0.00%) 0 (0.00%) 1 (9.09%) 
8 (21.62%) 1 (6.25%) 1 (10.00%) 6 (54.55%) 
16 (43.24%) 6 (37.50%) 6 (60.00%) 4 (36.36%) 
7 (18.92%) 4 (25.00%) 3 (30.00%) 0 (0.00%) 
4 (10.81%) 4 (25.00%) 0 (0.00%) 0 (0.00%) 
1 (2.70%) 1 (6.25%) 0 (0.00%) 0 (0.00%) 
0 (0.00%) 0 (0.00%) 0 (0.00%) 0 (0.00%) 
8 (21.62%) 3 (18.75%) 5 (50.00%) 0 (0.00%) 
29 (78.38%) 13 (81.25%) 5 (50.00%) 11 (100.0%) 
1 (2.70%) 0 (0.00%) 1 (10.00%) 0 (0.00%) 
2 (5.40%) 1 (6.25%) 0 (0.00%) 1 (9.09%) 
1 (2.70%) 1 (6.25%) 0 (0.00%) 0 (0.00%) 
1 (2.70%) 0 (0.0%) 1(10.00%) 00.00%) 
28 (75.68%) 12 (75.00%) 8 (80.00%) 8 (72.73%) 
4 (10.81%) 2 (12.50%) 0 (0.00%) 2 (18.18%) 
2 (5.40%) 0 (0.00%) 0 (0.00%) 2 (18.18%) 
32 (86.49%) 13 (81.25%) 10 (100.0%) 9 (81.82%) 
3 (8.11%) 3 (18.75%) 0 (0.00%) 0 (0.00%) 
37 (100.0%) 16 (100.0%) 10 (100.0%) 11 (100.0%) 


Procedures 


Our theoretical approach uses the EPIS framework to 
develop a focus group protocol (see below and the Appen- 
dix) to elicit information about the relevance and appropri- 
ateness of measuring implementation leadership, implemen- 
tation climate, and implementation citizenship in schools 
(Aarons et al., 2011). Focus groups were selected instead 
of other qualitative methods in order to allow synergistic 
discussion when individuals in the same focus group could 
elaborate on points articulated by other participants to pro- 
vide a deeper understanding of how the OIC constructs man- 
ifest in schools. This approach was appropriate given the 
time and financial constraints of the grant award that did not 
permit the use of individual interviews. Focus groups were 
held at the central office of each respective school district to 
provide a convenient and accessible location for participants. 

Separate focus groups were held for each stakeholder type 
(central office administrators, principals, and teachers) to 
remove obstacles to participant engagement (potential power 
differentials between stakeholders who may be in supervi- 
sory/managerial roles) as well as to allow for comparisons 
across stakeholder groups. The size of each focus group (6-9 
participants) was consistent with recommended procedures 
for thematic saturation (Guest, Bunce, & Johnson, 2006), 
with the exception of one principal focus group which only 
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had four participants because two had last minute conflicts 
that prevented them from attending the session. Each focus 
group session began with establishing norms and expecta- 
tions for the session, introductions, and an overview of con- 
structs (e.g., implementation leadership, climate, citizenship 
behavior) that were to inform the development of a set of 
school-based measures (Lyon et al., 2018). Prior to the focus 
group discussions, facilitators defined EBP and implemen- 
tation science for all participants. Participants had varying 
levels of understanding and experience with EBP implemen- 
tation. The focus group moderators situated the discussion 
of EBP with a focus on Tier 1 or universal programs that 
promote students’ social, emotional, or behavioral function- 
ing, which was the focus of the larger study (Lyon et al., 
2018). Participants also were provided an overview of each 
construct (definitions, how each construct is described in 
the literature) and asked to review existing measures (e.g., 
Implementation Leadership Scale, Implementation Climate 
Scale, and Implementation Citizenship Behavior Scale). Dis- 
cussion of each of the OIC constructs ensued after measure 
review, with specific probing questions designed to elicit 
feedback from the participants about each construct’s mani- 
festation in the school context. Each focus group was audio- 
and video-recorded and lasted 90-120 min. Participants 
were compensated $150 for their participation. 
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Measures 


We developed a systematic and comprehensive focus group 
protocol with questions that elicited open-ended responses 
from participants about: (1) implementation leadership (e.g., 
“What steps can leaders take to support their staff in suc- 
cessful implementation?’’); (2) implementation climate (e.g., 
“What specific aspects of implementation climate come to 
mind when you think about schools in which the adoption 
and implementation of evidence-based practices are high 
priorities for leaders and staff?’’); and (3) implementation 
citizenship behavior (e.g., “What is it about an employee in 
a school that allows him/her to support colleagues in imple- 
menting evidence-based practices effectively?’’). 


Data Analysis 


Focus groups were conducted by Lyon et al. (2018). Focus 
groups were transcribed and uploaded to NVivo QSR 10 
for data management. The EPIS model and OIC guided the 
development of the coding scheme. The coding scheme was 
developed using a rigorous, systematic, transparent, and iter- 
ative approach using the following steps. First, Lyon et al. 
(2018) independently coded two initial transcripts line-by- 
line to identify recurring codes. Second, they met as a group 
to discuss recurring codes and developed a codebook using 
an integrated approach to coding as certain codes were con- 
ceptualized during the focus group protocol development 
(i.e., deductive approach) and other codes were developed 
through a close reading of the two transcripts (i.e., inductive 
approach; Bradley, Curry, & Devers, 2007; Neale, 2016). 
Next, coders and principal investigators met to discuss and 
select common codes interpreted from the transcripts. The 
group collectively determined which codes were incorpo- 
rated into the final codebook. Then, operational definitions 
of each code were documented as well as examples of the 
code from the data, as well as when to use and not use the 
code. The coding scheme was applied to the data to produce 
a descriptive analysis of each code; the coding scheme was 
then refined throughout the data analytic process (Bradley 
et al., 2007). A table of codes and definitions is provided in 
the Appendix. Lyon et al. (2018) coded all data and over- 
lapped on 20% of randomly selected transcripts to determine 
inter-rater reliability. They met together on a weekly basis 
to discuss, clarify, verify, and compare emerging codes to 
ensure consensus. Agreement was calculated based on the 
number of words agreed upon; agreement between raters 
was excellent (percent agreement = 95.05% on parent codes 
and 98.60% on subcodes). 
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Results 


Participants’ conceptualizations of each of the constructs 
indicated unique elements of the school context that may 
be important for successful implementation of EBP to sup- 
port school mental health. However, there also were many 
similarities with the definitions and item content associated 
with the original development of the OIC measures in the 
implementation science literature. Overall, central adminis- 
trators and principals were broader in their descriptions of 
how the constructs manifest in schools, whereas teachers 
were more likely to focus on the specifics of EBP imple- 
mentation. See Table 2 for a list of codes and definitions 
and Table 3 for a list of OIC constructs, definitions, existing 
and new domains. Below we address each construct, noting 
which emerged as “new” and which previously “existed” in 
the original measures. 


New Dimensions 


Several new dimensions emerged from each of the OIC con- 
structs that are unique to the school context. First, Distrib- 
uted Leadership was identified within the implementation 
leadership construct. Participants from all three stakeholder 
groups discussed Distributed Leadership across multiple 
individuals (e.g., teacher leaders, career ladder teachers), 
teams (school psychologists, counselors, teacher leaders, 
teacher mentors/coaches), and levels (schools and central 
office) to support EBP implementation. Overall, many par- 
ticipants from each of the three stakeholder groups chal- 
lenged the notion that leadership in schools is centralized 
at the school administrator level and highlighted the impor- 
tance of a team-based leadership approach. One principal 
said, “There are a lot of different leaders in the building... 
the principal might not know everything about EBP, so we 
rely on our experts in the building to know those answers.” 
A central administrator highlighted that “the building lead- 
ership team develops a plan [for EBP implementation] ver- 
sus the school administrator [often] establishes clear school 
standards and accountability measures for implementation.” 
Although the principal often is seen as the primary leader in 
a school, the general sentiment across all three stakeholder 
groups was that leadership teams (which comprise several 
stakeholders in school settings) share the responsibilities of 
supporting EBP implementation. 

Second, Existing Supports to Deliver EBP was a new 
dimension of implementation climate that emerged across 
all participants that captured how existing school structures 
or resources could be incorporated or repurposed to sup- 
port EBP implementation and build capacity across levels 
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Table 2 Definitions of codes 


Codes 


Implementation leadership: Refers to leadership that involves specific 
actions that are performed to support the adoption and use of EBP in 
the school setting. This can include proactive leadership, support- 
ive leadership, knowledgeable leadership, perseverant leadership, 
and available leadership. This also includes comments about how 
leadership is distributed and built within a school, how leaders are 
held accountable across all stakeholders, or the level of leadership 
necessary for successful implementation and sustainment 


Implementation climate: Climate refers to educators’ share perception 
of the importance of EBP implementation. This includes Focus on 
EBP, Educational Support for EBP, Recognition for EBP, Rewards 
for EBP, Selection for EBP, and Selection for Openness. Comments 
that would be coded under this includes educators’ perceptions of 
norms/expectations of implementation, the existing structures and 
resources in place to support implementation, incentives to imple- 
mentation, school goals/priorities, the varying demographic levels of 
students and staff across schools. 
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Subcodes 


Proactive leadership: Involves the extent to which a leader establishes 
clear standards surrounding implementation, develops plans to facili- 
tate implementation, and removes obstacles to implementation 

Supportive leadership: Involves the degree to which leaders support 
employee efforts to learn more about or use EBP, and recognizes and 
appreciates employee efforts 

Knowledgeable leadership: Occurs when employees believe that leaders 
“know what they are talking about” surrounding EBP and are able to 
answer questions effectively 

Perseverant leadership: Refers to the extent to which leaders carry on 
through challenges of implementation and react to critical issues sur- 
rounding implementation 

Available leadership: Refers to the extent to which leaders are acces- 
sible when it comes to implementation, make time to meet about 
implementation, are available to discuss implementation or provide 
help, and can be contacted with problems or concerns 

Distributed leadership: Comments about Distributed Leadership 
structures in schools, especially as they relate to implementation. May 
include comments about leadership teams, how principals facilitate 
Distributed Leadership. Distributed Leadership also includes the level 
of leadership that is most relevant to implementation and accountabil- 
ity across those levels. These comments refer to the different grade- 
level teams/leaders within teachers to be accountable for implement- 
ing and sustaining new practices 


Focus on EBP: References whether the organization thinks implemen- 
tation is important, a top priority, or has effective use of EBP as a 
primary goal 

Educational support for EBP: Refers to whether the organization sup- 
ports EBP training, staff travel to conferences or workshops, or sup- 
plies training materials or other supports (e.g., journal articles) 

Recognition for EBP: Refers to whether the organization views staff 
with EBP experience as experts, holds them in high esteem, and is 
likely to promote them 

Rewards for EBP: Rewards are financial incentives for the use of EBP, 
whether staff that use EBP are more likely to get bonuses/raises, or 
accumulated compensated time. This code also includes comments 
about contextually appropriate incentive structures in schools that 
are driving motivators for educators. May include student outcomes, 
leadership opportunities, protected time 

Selection for EBP: Refers to the extent to which the organization prefers 
to hire staff that are flexible, adaptable, or open to new interventions 

Selection for openness: Refers to the extent to which the organiza- 
tion prefers to hire staff that are flexible, adaptable, or open to new 
interventions 

Existing supports to deliver EBP: Existing school structures or 
resources that could be incorporated/repurposed to support implemen- 
tation at an organizational level. May include comments about existing 
teams or staff interactions relevant to implementation, structures for 
professional development (e.g., individualized professional develop- 
ment plans) 

Prioritization of EBP: Comments about school priorities, policies and 
procedures, and how they align with the implementation of social, 
emotional, and behavioral programming 
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Table 2 (continued) 


Codes 


385 


Subcodes 


Implementation citizenship behavior: Citizenship refers to the different Helping others: Refers to the extent to which educators assist others 


elements of citizenship behaviors or the degree to which educators 
go “above and beyond” to support implementation. Demonstrates 
a commitment to EBP by keeping informed about the EBP being 
implemented and supporting colleagues to meet EBP standards. 
Includes educators’ willingness to share knowledge with peers 

or their community, opening their classroom for observation and 
feedback. 


to make sure they implement EBP, help teach EBP, implementation 
procedures, or help others with responsibilities related to EBP (e.g., 
completing fidelity assessments) 


Keeping informed: Refers to whether educators keep up to date on 


changes in EBP policy, follow the latest news or new findings regard- 
ing EBP, and keep up with agency communications related to EBP 


Information sharing: Educators demonstrating citizenship by sharing 


knowledge and information with peers and beyond the school setting 
(e.g., connecting with parents, and community members) to further 
the implementation effort; also includes be ability and openness to 
take that knowledge and information and adopt it into practice 


Observation/feedback: Educators demonstrating citizenship by offer- 


ing to observe, be observed in the classroom, and provide or receive 
feedback on their professional practice 


Table 3 Original OIC construct definitions, existing domains, and added domains 


Original OIC construct Definition 


Strategic implementation leadership Strategic implementation leadership is 
a subcomponent of general leadership 
that involves specific behaviors that 
support or inhibit implementation in 
service organizations. These include 
leaders being knowledgeable and 


Existing domains New domains 
Proactive leadership 
Supportive leadership 
Knowledgeable leadership 
Perseverant leadership 
Available leadership 


Distributed leadership 


able to articulate the importance of 
implementation and being supportive 
of staff, proactive in problem solving, 
and perseverant in the implementation 


process 


Strategic implementation climate Strategic implementation climate 


encompasses employee perceptions 
of the organizational supports and 
practices that help to define norms 


and expectations with regard to 


the implementation of new EBP. 


Focus on EBP 

Educational support for EBP 
Recognition for EBP 
Rewards for EBP 

Selection for EBP 

Selection for openness 


Existing supports to deliver EBP 
Prioritization of EBP 


A positive implementation climate 
signals what is expected, supported, 
and rewarded in relation to use of pro- 


grams or practices 


Implementation citizenship behavior Citizenship behaviors are exhibited 


when employees go “above and 


beyond” their core job aspects or 


Helping others 
Keeping informed 


Information sharing 
Observation/feedback 


standard “‘call of duty” to further the 
mission of the organization. Imple- 
mentation citizenship behaviors are 
those that demonstrate a commitment 
to EBP by keeping informed about 


the EBP being implemented and 


supporting colleagues to meet EBP 


standards 


(individual teacher/staff, school and central office). All par- 
ticipants mentioned using existing school teams (i.e., guid- 
ance teams, school improvement planning teams, career 
ladder teachers, grade-level teams, professional learning 


communities) that are naturally embedded in schools’ exist- 
ing infrastructure to play a role in the adoption, implementa- 
tion, and sustainment of EBP in schools. One central admin- 
istrator commented that “there are routines or structures 
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established within the school [such as] coordinated profes- 
sional learning communities or planning time...that com- 
municates with people and [trains] new staff to make [EBP 
implementation] more feasible” in schools. The reliance on 
existing school teams may facilitate EBP use and champion 
implementation efforts forward on multiple levels. Another 
aspect of Existing Supports endorsed by all stakeholder 
groups was the use of peer-to-peer relationships and infor- 
mal social interactions among teachers/staff to support EBP 
implementation efforts by reducing feelings of isolation that 
are common in schools. 

Third, Prioritization of EBP also was a new dimension 
of implementation climate created to capture district and 
school priorities, policies and procedures, and their align- 
ment with EBP implementation. Participants varied in their 
description of Prioritization of EBP. Some central admin- 
istrators remarked that Prioritization of EBP often is based 
on district-level support and resources (i.e., funding) or dis- 
trict priorities, even though there is a “disconnect between 
what the district prioritizes and what actually happens [in 
practice].” Principals elaborated on the challenge of compet- 
ing priorities in academic versus social-emotional EBP that 
teachers/staff manage that often leads to EBP fatigue. One 
principal said that “we are so consumed with a lot of other 
academic frameworks that we would like to get to the social- 
emotional learning standards...but, they are not a prominent 
feature.” Both principals and teachers expressed the need for 
central office and school leadership to de-prioritize compet- 
ing EBP and other initiatives more broadly and suggested 
that “clearly defined” school priorities around EBP and ini- 
tiatives is essential to gain buy-in from teachers/staff. 

Fourth, Information Sharing was a new dimension that 
emerged as part of implementation citizenship behavior. The 
majority of principals and teachers identified Information 
Sharing in their discussion of implementation citizenship 
behavior where school personnel actively shared knowledge 
(e.g., expertise, tips, strategies) with their colleagues, peers, 
and broader community (parents). Principals also discussed 
that it was “above and beyond” behavior to “put them[selves] 
out there” to “share knowledge or [their] expertise with oth- 
ers within [their] building as well as within [their] district or 
beyond.” Observation/Feedback also was a new dimension 
of implementation citizenship. Most participants described 
this construct as willingness to open up their practices for 
observation and feedback as well as provide opportuni- 
ties for educational site visits from other schools/districts 
to observe their practices. One central administrator noted 
that “having [staff] that [implement EBP well] model those 
practices for other staff and colleagues” is critical to this 
construct. 
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Existing Dimensions 
Implementation Leadership 


Comments from some central administrators and princi- 
pals aligned with Aarons et al. (2014a) definition of pro- 
active leadership. Many central administrators and princi- 
pals described proactive leadership in terms of collecting 
multiple sources of data to identify growth and areas of 
improvement as well as track accountability. One central 
administrator indicated “using data to show where there’s 
growth and where there’s room for improvement” would be 
helpful in increasing EBP use. Some central administrators 
and principals described supportive leadership similarly 
to Aarons et al. (2014a) by saying that leaders were: (1) 
present and fully engaged in implementation efforts; and 
(2) able to provide resources (e.g., time, physical space, 
embedded coaching, behavioral observations, materials). 
One novel component described by central administrators 
and teachers was showing support by helping to “prioritize 
or de-prioritize” competing demands or initiatives in shared 
decision making given the EBP fatigue or overload that often 
exists in schools. Some teachers differed in their depiction of 
supportive leadership and expressed that supportive leader- 
ship entails a leader who: (1) is willing to learn about EBPs 
alongside teachers and implementation agents (e.g., “I would 
like to see administrator willingness to get involved in the 
learning with [teachers, where they] come to [the] classroom 
and do the strategy...”); (2) understands and has empathy 
for the implementation challenges that teachers face; and (3) 
is intentional about creating structures (e.g., check-ins with 
whole staff and individuals, troubleshooting) to ensure EBP 
implementation is prioritized. 

Similar to Aarons et al. (2014a) definition, central admin- 
istrators and teachers characterized knowledgeable leader- 
ship as providing “the why” to teachers and staff to increase 
understanding of the expectations, needs, and goals of EBP 
implementation. Many principals noted additional aspects 
of knowledgeable leadership including the leader’s ability 
to “consistently model EBP” and the ability to discern and 
use credible “evidence to support decision making to use 
EBP.” Perseverant leadership was infrequently discussed 
across the participants. Most principals described persever- 
ant leadership as active involvement with teacher and school 
staff implementation efforts to create an environment where 
“EBP implementation carries on even in the absence of the 
principal.” Consistent with Ehrhart et al. (2018), central 
administrators and teachers discussed available leadership 
in terms of being open and present to provide help and 
feedback, while principals did not mention this construct. 
Teachers underscored the importance of an available leader 
as someone they can trust to have an “open door to bring 
up concerns” to enhance implementation or barriers to 


School Mental Health (2019) 11:379-399 


implementation. One teacher specifically noted that available 
leadership entails “physical availability... actually having 
[their] door open or being in the classroom and... emotional 
availability [where they are] willing to address something.” 


Implementation Climate 


Many principals mentioned having a “building focus” on 
EBP as well as setting “explicit goals around [adoption of] 
EBP” among staff carrying out implementation. Similar to 
Ehrhart et al. (2014), participants described educational sup- 
ports for EBP in the context of professional development 
trainings (i.e., workshops, seminars, conferences) that are 
initiated and supported at the district level across all schools. 
One administrator mentioned, “there is an underlying 
assumption that everybody understands what an EBP is... 
[but she] never [uses the term] EBP because people would 
freak out...and assume that it is a major effort to imple- 
ment.” Many teachers discussed the lack of educational sup- 
port in terms of tangible materials. One teacher noted that 
she has “gone to training, [but] if [she] wants to implement 
[the EBP, she] would have to buy it or have the materials to 
be able to copy and paste all the stuff.” Additional aspects 
of Educational Support for EBP that many principals and 
teachers brought up included having: (1) access to experts in 
the field available to teachers and school staff; and (2) pro- 
tected time to reflect, adapt, develop implementation plans, 
and integrate EBP into daily practice. 

Recognition for EBP was infrequently discussed. The 
only component of the existing subscale that participants 
noted was the potential for promotion. One principal 
described “greater access to leadership opportunities” as 
compared to “traditional promotion” as recognition oppor- 
tunities in schools. All participants agreed and stated that 
traditional Rewards for EBP (i.e., in the form of promo- 
tions or financial incentives) were not appropriate or feasible 
within the school context. Central administrators explained 
how “financial incentives [like] bonuses and raises are off 
the table...[due to] union contracts and involvement.” Fur- 
thermore, some principals specifically noted that financial 
incentives are “not a motivator” and “will divide [them] as 
a team” since “the heart of why [school staff] do what [they] 
do...” is to see “[their] kids succeed.” Interestingly, teach- 
ers articulated that they “get so much more [from] being 
compensated for time to plan” around EBP implementation, 
which would be the most favorable reward that they could 
receive over and above monetary incentives. Selection for 
EBP was mentioned twice and only among teachers. Teach- 
ers expressed the lack of understanding among school staff 
as to how new hires are selected. One teacher said, “Unless 
[school staff] are on the hiring committee, [they] do not 
know why a person is selected.” The reasons that some peo- 
ple are selected and hired into their roles appear to be largely 
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unknown to teachers in participating schools. Selection for 
Openness was not discussed. 


Implementation Citizenship Behavior 


Participants’ conceptualization of Helping Others was simi- 
lar to Ehrhart et al. (2015) definition with respect to respon- 
sibilities for EBP implementation and helping teach EBP 
implementation procedures; however, none of the partici- 
pants described helping to ensure proper EBP implemen- 
tation (monitoring fidelity). Most frequently across par- 
ticipants, Helping Others was described in the context of 
teamwork, “equally sharing the workload among staff” via 
formal (e.g., professional learning committees, grade group 
meetings, demonstration teachers) and informal methods 
(e.g., distributing responsibilities, mundane daily tasks). 
Keeping Informed was not frequently discussed. Only one 
component of the Keeping Informed subscale from Ehrhart 
et al. (2015) definition, latest news regarding EBP, was dis- 
cussed among teachers. Central administrators described 
“attending trainings [and professional development] as a key 
indicator” of Keeping Informed as well as keeping up with 
the “latest news [such as] reading online sources” wherever 
available during personal off-work times. However, district- 
or school-wide communication related to EBP and changes 
in EBP policies and procedures were not mentioned. 


Discussion 


Adaptation and application of existing constructs and 
corresponding measures of the OIC to the school setting 
provide promising opportunities to advance school-based 
implementation science and practice. This study extends 
research led by Aarons et al. (2014a) and Ehrhart et al. 
(2014, 2015, 2018) to articulate key aspects of the OIC of 
schools that relate to EBP implementation success in two 
local school districts in the Northwestern USA. See Fig. 1. 
Specifically, this study engaged key education stakeholders 
(central administrators, principals, and teachers) to gather 
information regarding their perceptions of the appropri- 
ateness and conceptual boundaries of the OIC constructs 
(implementation leadership, climate, and citizenship behav- 
ior) in schools. Three general findings emerged from our 
focus groups. First, we found critical differences unique 
to the education sector and several similarities with other 
service sectors (e.g., community mental health and child 
welfare) in how OIC constructs manifest, which are further 
discussed below. Second, all stakeholder groups indicated 
that greater attention to the OIC constructs offers a promis- 
ing approach to improving youth mental health outcomes 
through the adoption and delivery of higher quality school- 
based mental health services and supports. Last, the OIC 
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Fig. 1 School organizational implementation context (OIC) concept map. The light gray bars depict new dimensions of the OIC constructs in 


schools 


constructs appear to have potential utility across each stage 
of the implementation process (exploration, preparation, 
implementation and sustainment; Aarons et al., 2011). Based 
on the data gleaned from this study, we offer recommenda- 
tions to researchers and community stakeholders who may 
be interested in examining these organizational constructs 
in schools. 

We note that many dimensions of the OIC were not fre- 
quently discussed that may play a critical role in successful 
EBP implementation in schools. For example, perseverant 
leadership may be one aspect of implementation leadership 
that warrants more attention in schools since implementa- 
tion is a long and arduous process that often takes 2-5 years 
or more for full implementation to occur (Fixsen, Naoom, 
Blasé, Friedman, & Wallace, 2005). A lack of perseverant 
leadership may contribute to the number of EBP implemen- 
tation efforts that fail in schools (Gottfredson & Gottfred- 
son, 2001), and the frequently encountered “flavor of the 
month” phenomenon in which organizations get stuck in a 
cycle of taking on and abandoning new programs and prac- 
tices (Basch, Sliepcevich, Gold, Duncan, & Kolbe, 1985). 
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Schools should carefully consider the ways to problem 
solve implementation challenges prior to decommissioning 
an EBP, which may involve the de-implementation of other 
ineffective programs and practices (Wang, Maciejewski, 
Helfrich, & Weiner, 2017). In addition, participants in the 
current study augmented implementation leadership with an 
additional dimension: Distributed Leadership, where lead- 
ership is shared across multiple individuals and roles in a 
school as well as the central office (Angelle, 2010). This 
dimension captures shared leadership responsibilities that 
are held across stakeholders in schools. Although there also 
was recognition among participants that principals tend to 
be the ultimate arbiters of most school-related issues, there 
may be various leaders that spearhead different tasks or con- 
tent areas in schools that embody middle leaders or middle 
managers, and implementation efforts ought to consider the 
critical role that multiple leaders may play in schools (Priest- 
land & Hanig, 2005; Birken et al., 2015). 

Our results suggest that implementation climate may 
manifest somewhat differently in schools. While some com- 
ponents (Focus on EBP; Educational Support for EBP) were 
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similarly conceptualized to more traditional allied health- 
care delivery settings, other aspects of implementation cli- 
mate differed. First, it was clear that selection for EBP and 
selection for openness may not be feasible components of 
implementation climate measurement in schools, as some 
stakeholders at the school building level often do not have 
influence in the hiring process to select staff with a proclivity 
for use of EBP. It is concerning that teachers described the 
selection and hiring process as lacking transparency. Our 
participants often indicated they were not aware of the dis- 
trict standards for hiring new employees, which may indi- 
cate that hiring standards are not readily available in the 
districts in which this work was conducted, or they do not 
exist. Without transparent processes, it is more difficult to 
make changes that might support the recruitment and selec- 
tion of individuals with more openness, knowledge, skills, 
and abilities related to the use of EBP. Second, our find- 
ings suggest that the reward and recognition components of 
implementation climate become increasingly blurred since 
there is universal agreement that financial rewards are inap- 
propriate and unfeasible in schools (Lyon et al., 2018) or, 
that they may receive less emphasis compared to the poten- 
tial benefits of EBPs for students. These results are consist- 
ent with Lyon et al. (2018), who quantitatively examined 
implementation climate in schools and found that the fac- 
tor structure of most implementation climate subscales was 
upheld but there was less evidence supporting the rewards 
subscale. However, consistent with one item in that subscale, 
our stakeholders posited non-financial incentives—such as 
the provision of protected time for planning or additional 
preparatory periods—as potential recognition or reward for 
EBP use, which central offices and schools may consider to 
develop positive implementation climates. Third, two new 
dimensions (Existing Supports to Deliver EBP and Prior- 
itization of EBP) emerged that also may capture important 
aspects of implementation climate in schools. Given the 
financial constraints that many schools face, stakeholders 
discussed the importance of capitalizing on existing school 
structures and supports and leveraging existing structures 
(teams, personnel) to maximize limited resources to facili- 
tate EBP implementation. This may help mitigate resource- 
related barriers to implementation. Similarly, stakeholders 
identified Prioritization and De-Prioritization of EBP and 
other initiatives as an important component of implemen- 
tation climate. Schools often experience significant EBP 
fatigue and prioritizing certain implementation efforts may 
increase the likelihood of successful EBP adoption, use, and 
sustainment. Lastly, OIC constructs may be malleable deter- 
minants in schools that act as mechanisms through which 
organizationally focused implementation strategies may 
impact implementation outcomes. This conceptualization 
mirrors the direction of increasing implementation research 
focused on developing a more robust understanding of how 
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implementation strategies exert their effects (Lewis et al., 
2018). With proper implementation strategies deployed, it 
may be possible to improve these aspects of implementa- 
tion climate and improve EBP use in schools (Cook, Lyon, 
Locke, Waltz, & Powell, submitted; Lyon, Cook, Locke, 
Powell, & Waltz, submitted). 

Similar to implementation climate, our results suggest 
that aspects of implementation citizenship behavior (Helping 
Others and Keeping Informed) were comparably conceptual- 
ized in schools; however, there also were unique differences 
that stakeholders highlighted for implementation citizen- 
ship behavior in schools. The Helping Others and Keep- 
ing Informed dimensions were mostly present in schools 
with three notable differences—none of the stakeholders 
discussed: (1) ensuring proper EBP implementation; (2) 
district- or school-wide communication related to EBP; or 
(3) changes in EBP policies and procedures, which suggests 
these behaviors are rare, not salient, or nonexistent in school 
settings. These may be important behaviors to cultivate to 
support EBP implementation in schools. In addition, two 
new subconstructs of implementation citizenship emerged: 
Information Sharing and Observation/Feedback. Stakehold- 
ers highlighted the necessity of Information Sharing with 
their colleagues, particularly sharing their expertise with 
others in their building and beyond. Interestingly, stakehold- 
ers also noted the value of observation and feedback—both 
opportunities to provide and receive Observation/Feedback 
on their performance. These additions to implementation 
citizenship may be critical practices in the education sector. 
Strategies to improve opportunities for collaboration and 
feedback may help strengthen implementation citizenship 
in schools. Moreover, identifying and supporting key opin- 
ion leaders (Atkins et al., 2008) and EBP champions (Aitken 
et al., 2011) may reflect social strategies to influence imple- 
mentation citizenship behaviors that facilitate EBP uptake, 
delivery, and sustainment. However, the extant literature 
offers little guidance on strategies to promote implementa- 
tion citizenship in schools. 

The results of our study also were consistent with the 
implementation leadership dimensions (proactive, sup- 
portive, knowledgeable, perseverant, and available) found 
in the broader implementation science literature (Aarons 
et al., 2014a; Ehrhart et al., 2018). However, it appears that 
some dimensions were more salient and prevalent (support- 
ive) in schools than others (perseverant). Participants noted 
that supportive leadership encompassed the provision of 
tangible resources (materials) as well as emotional support 
for EBP use. Interestingly, participants also discussed Pri- 
oritization and De-Prioritization of EBP and other school 
initiatives to prevent EBP fatigue as supportive leadership. 
This is an important point to consider as schools often face 
barriers to implementation that include inadequate time, 
energy, effort, and resources for implementation (Forman, 
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Olin, Hoagwood, Crowe, & Saka, 2009; Locke et al., 2015). 
Indeed, in a recent school-focused adaptation of the Expert 
Recommendations for Implementing Change (ERIC) com- 
pilation of implementation strategies in health care (Powell 
et al., 2015), Cook et al. (submitted) added “pruning compet- 
ing initiatives” to address exactly this issue. 


Limitations and Future Directions 


While this study explored the qualitative nuances of imple- 
mentation leadership, climate, and citizenship behavior in 
schools, several limitations are noted. First, these results 
should be interpreted with caution as the data only were 
gathered in two predominantly white school districts in the 
Northwestern USA, which may limit the geographic gener- 
alizability of the results to areas with similar racial/ethnic 
diversity and/or socioeconomic status as well as school char- 
acteristics. Second, the teachers who participated were not 
self-nominated, but were nominated by the district’s central 
administrators. Third, the sample was predominantly female, 
which may limit the interpretation of the results. Fourth, the 
current project was focused primarily on Tier 1 universal 
supports. The OIC constructs investigated may manifest dif- 
ferently for more intensive social, emotional, or behavioral 
supports. Lastly, this study only focused on the OIC con- 
structs, and no individual-level constructs were explored, 
which also is important for EBP implementation (Becker- 
Haimes et al., 2017; Feuerstein et al., 2017). 


Conclusion 


Findings from this study support the assertion that the three 
OIC constructs (implementation leadership, climate, and 
citizenship) are relevant and applicable to the school context. 
Results also demonstrate the utility of qualitative methods 
to inform revisions to existing constructs from other service 
settings (e.g., health care, child welfare, juvenile justice) 
with the goal of facilitating application in novel settings, 
such as schools. As noted earlier, the OIC constructs also 
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have been codified in a set of established organizational eval- 
uation measures (Lyon et al., 2018), which may be further 
revised based on the information collected herein. Next steps 
could include validation of the expanded versions of OIC 
constructs in the context of school-based implementation 
efforts. Use of qualitative methods to adapt existing meas- 
ures is one way to avoid the “home-grown measure” phe- 
nomenon in which measures are rapidly created “in house 
to assess a construct in a particular study sample without 
engaging in proper test development procedures” (Martinez, 
Lewis, & Weiner, 2014, p. 4). This phenomenon has recently 
plagued implementation science and diminished the com- 
parability of results across studies. Home-grown measures 
often are highly specified to the population in which the 
measure was used and have limited generalizability and util- 
ity for comparisons across studies (Martinez et al., 2014). 
Additionally, there is a need for research that carefully con- 
siders how the OIC constructs manifest across individuals 
at different levels (e.g., individual provider, school/organi- 
zational, and central office/district) and how to design spe- 
cific implementation strategies that precisely target specific 
aspects of OIC constructs in schools. 
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Appendix 


Structure to OASIS Study 1 - Focus Group Session 


Brief presentation on purpose of the focus groups (5 minutes) 


Introduction to Evidence-Based Practice and Implementation Science to provide context 
(5-10 minutes) 


e Brief overview of evidence-based practices in the schools 


fe) 


“Before we jump into the focus group itself, we want to provide a little context for 
our discussion. This entire project is about the implementation of evidence-based 
practices in the schools. In particular, we are most interested in Tier I or 


universal programs that promote students’ social, emotional, or behavioral 


functioning. 
= We define evidence-based practices as clearly-specified programs, 


practices, and supports with demonstrated empirical support for their 
efficacy/effectiveness and application to target groups. 

* “What does this mean to you?” 

Next consider: What is the opposite of evidence-based practices?” 

" > Lead a discussion about why evidence-based practices have been 
established and why we have reached a point where legislation and policy 
emphasize the increased adoption and delivery of evidence-based practices 
in the school setting. 

“A key part of evidence-based practices are the active ingredients of the 
prevention or intervention program, Think about evidence-based practices as a 
good cooking recipe. If one of the ingredients is left out, such as flour or sugar 
left out of chocolate chip cookies, then one won't end up with chocolate chip 
cookies. Evidence-based practices can be thought of similarly. There are core 
ingredients or components of an intervention that need to be adopted in order for 
them to produce the findings demonstrated in research studies. The trouble is that 
when attempting to transfer evidence-based practices in to the school setting, 
active ingredients are often not fully implemented or insufficiently adopted 
school-wide to produce desired outcomes.” 


e Explanation of school-based implementation science 


fe) 


“One of the most significant issues we face is this implementation gap, which 
reflects the discrepancy between science and practice. It turns out that research 
findings don’t crawl out of journals and make their way into the everyday settings 
on their own. 

“Implementation science is the study of methods to promote the integration of 
research findings and other evidence-based practices into routine practice. We 
want to do this so children can actually benefit from them. 

Research has uncovered several factors — such as supportive leadership — that 
promote the use of evidence-based practices in the school setting. The OASIS 
project is about measuring key aspects of the school setting that impact the 
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successful use of evidence-based Tier I practices for students’ social, emotional, 
and behavioral functioning.” 


Importance of organizations to successful implementation (5 minutes) 


Description of the organizational implementation context (OIC) 


“Your handouts include this overview figure and a description of each of our key constructs” 


o “Our project focuses on ways to measure the organizational implementation 
context, which refers to aspects of the school setting that are closely related to 
actual implementation. 


= For example, leadership practices that create an environment that is 
conducive to — or problematic for — the implementation of new 
innovations is a core component of the OIC. In addition, a teacher’s 
attitudes toward evidence-based practices are likely to impact their 
motivation and commitment to implement those practices once the teacher 
has received training.” 


o What the OIC isn’t_/ what we aren’t talking about 

= “We recognize that there is wide variation in the extent to which 
professional development / implementation supports are used effectively 
when introducing new programs. We are most interested in 
organizational factors that may influence implementation success whether 
or not professional development supports are optimal.”’ 

= “We are most focused on the factors that influence whether programs are 
implemented with fidelity and have their intended effects on students.” 


o Adaptation and refinement of existing measure that capture critical aspects of the 
OIC 

= “This project benefits from previous work conducted in Child Welfare 
and Youth Mental Health, where measures assessing aspects of the 
organizational implementation context were developed and validated. 
Our project seeks to adapt and refine these measures for use in the school 
setting so we can better understand whether a given school is ready to 
begin implementing or assess key variables during active implementation 
that serve as barriers to success.” 

=" “There are four measures assessing unique, but related, components of 
the organizational implementation context. These are the Implementation 
Climate Scale, Implementation Leadership Scale, Implementation 
Citizenship Behavior Scale, and Evidence-Based Practice Attitudes 
Scale. Each of the measures captures specific aspects of the 
organizational implementation context that helps explain why 
implementation is likely to be successful within a given school or district. 
Each measure will be described in more detail later.” 
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" “Ultimately, the data from these measures are intended to provide 
actionable information that will enable district personnel, external 
consultants, or implementation coaches to pinpoint specific 
implementation supports that can facilitate the adoption and delivery of 
evidence-based practices that are being rolled out in the school. 

" “Does anybody have any questions or comments?” 


Strategic Implementation Climate 


e Present the Strategic Implementation Climate construct and key elements of the construct 
(provide a handout that outlines the construct and bullet points key idea associated with 
the construct) (5 minutes) 


o Recall that we are most interested in Tier 1 or universal programs that promote 
students’ social, emotional, or behavioral functioning 


o “First, we are going to discuss Strategic Implementation Climate, as measured 
by the Implementation Climate Scale.” 


o “Strategic Implementation Climate is the staff's shared perception of the 
importance of EBP implementation and includes staff perceptions of norms and 
expectations with regard to implementation. 

= SIC differs from general school climate, which refers to how individuals 
within the system collectively experience the school as supportive. 
Implementation climate is therefore a sub-component of general climate 
that reflects people’s perceptions about the extent to which the school 
supports taking on and implementing new things.” 

o ©6“Schools can range from an open and positive implementation climate to a 
closed and negative implementation climate. The specific aspects of Strategic 
Implementation Climate evaluated by the Implementation Climate Scale include 
Focus on EBP, Educational Support for EBP, Staff Recognition for EBP, Rewards 
for EBP, Staff Selection for EBP and Staff Selection for Openness.” 

o “Focus on EBP references whether the organization's thinks implementation is 
important, a top priority, or has effective use of EBP as a primary goal.” 

o “Educational Support for EBP refers to whether the organization supports EBP 
training, staff travel to conferences or workshops, or supplies training materials 
or other supports (e.g., journal articles).” 

o “Recognition for EBP refers to whether the organization views staff with EBP 
experience as experts, holds them in high esteem, and is likely to promote them.’ 

o “Rewards for EBP are financial incentives for the use of EBP, whether employees 
who use EBP are more likely to get bonuses/raises, or accumulated compensated 


, 


time. 

o “Selection for EBP refers to the extent to which the organization prefers to hire 
employees who have previously used EBP, have formal education supporting 
EBP, and value EBP.” 

o _“Selection for Openness refers to the extent to which the organization prefers to 
hire employees who are flexible, adaptable, or open to new interventions.” 
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o DO YOU HAVE ANY QUESTIONS ABOUT STRATEGIC IMPLEMENTATION 
CLIMATE OR ITS SUBCONSTRUCTS? 


SIC ACTIVITY 


Part 1 — Individual reflection on SIC (1-2 minutes): 
=> GENERATE IDEAS: “First, we would like to give you each I minute to think 
privately about the kinds of ideas on a blank piece of paper about what 
information should be included in a measure of SIC. This might be questions that 
you think would be important to ask or specific behaviors or perceptions that we 


should be sure to capture. Even though the measure is included in your packet, we 
ask that you refrain from reviewing it for the time being.” 


Part 2 — Review of original and adapted ICA measures (15-25 minutes): 
=> MEASURE REVIEW: 

» “Next, we would like you to review the document in your folder that lists 
the original items from the Implementation Climate Scale in one column, 
our adapted version of the item in the next column, and a third column for 
your notes/comments. The form also asks for your ratings of item 
appropriateness to schools. After sifting through all the items, we would 
also like you to reflect on additional items that we should consider 
including that would fall under the Implementation Climate Scale. These 
items may include ideas you generated during the first part of this 
activity.” 

« “We will be collecting these forms at the end of the focus group” 

e Provide 5-10 minutes for review and 10-15 minutes for 
discussion. 

" Facilitate discussion (potential probes): 

e “What feedback do you have on the measures and their ability to 
assess Strategic Implementation Climate in Schools?” 

e “What are we potentially missing with this measure that could be 
included to provide for a more valid and useful measure of 
implementation climate?” 

e “Beyond general school climate, such as how much people feel 
included or cared about by the organization, what specific aspects 
of climate come to mind when you think about schools in which the 
adoption and implementation of evidence-based practices are high 
priorities for leaders and staff?” 

e How would you know if your school or district values EBP? 

e What are the various aspects of school environments that send the 
message that EBP adoption and implementation are valued in this 
school? 
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e What policies, practices, procedures, or systems would you expect 
to see in place in a school that prioritizes EBP implementation? 

e “What indicators do you believe would capture a school building 
that has a positive implementation climate that would likely lead to 
successful evidence-based practice implementation?” 


Strategic Implementation Leadership 


e Present the Strategic Implementation Leadership construct and key elements of the 
construct (provide a handout that outlines the construct and bullet points key idea 
associated with the construct) (5 minutes) 

o ©6“Next, we are going to discuss strategic implementation leadership, which is 
measured by the Implementation Leadership Scale.” 

o © “Strategic implementation leadership refers to a specific form of leadership that 
involves actions that are performed to support the adoption and use of an 
innovative program or practice being introduced into an organization. It differs 
from general leadership qualities that may help to promote a positive work 
climate and motivate staff to stay on with an organization, but do not capture 
things that leaders can do to support implementation specifically. 

o For example, strategic implementation leadership captures behaviors that fall 
under five different categories of behavior: Proactive Leadership, Supportive 
Leadership, Knowledgeable Leadership, Perseverant Leadership, and 
Availability.” 

o Proactive Leadership involves the extent to which a leader establishes clear 
standards surrounding implementation, develops plans to facilitate 
implementation, and removes obstacles to implementation. 

o Supportive Leadership involves the degree to which leaders support employee 
efforts to learn more about or use EBP, and recognizes and appreciates employee 
efforts. 

o Knowledgeable Leadership occurs when employees believe that leaders “know 
what they are talking about” surrounding EBP and are able to answer questions 
effectively. 

o Perseverant Leadership refers to the extent to which leaders carry on through 
challenges of implementation and react to critical issues surrounding 
implementation. 

o Available Leadership refers to the extent to which leaders are accessible when it 
comes implementation, make time to meet about implementation, are available to 
discuss implement or provide help, and can be contacted with problems or 
concerns. 


SIL ACTIVITY 


e Part 1 — Individual reflection about SIL (1-2 minutes): 
=> GENERATE IDEAS: “First, we would like to give you each 1 minute to think 
privately about the kinds of ideas on a blank piece of paper about what 
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information should be included in a measure of SIL. [This might be questions that 
you think would be important to ask or specific behaviors or perceptions that we 
should be sure to capture. Even though the measure is included in your packet, we 
ask that you refrain from reviewing it for the time being].” 
e Part 2 — Review original and adapted ILS measures (15-25 minutes): 
> MEASURE REVIEW: Distribute original and adapted ILS measure form. Have 
each member identify words that may be confusing, items that seem irrelevant for 
the school context, and additional items that seem relevant to capture under the 
measure 
= “Next, we would like you to review the document in your folder that 
provides the original items from the Implementation Leadership Scale in 
one column, our adapted version of the item in the other column, and a 
third column for your notes/comments. The form also asks for your ratings 
of item appropriateness to schools. After sifting through all the items, we 
would also like you to reflect on additional items that we should consider 
including that would fall under the Implementation Leadership Scale. 
These items may include ideas you generated during the first part of this 
activity.” 
= “We will be collecting these forms at the end of the focus group” 
e Provide 5-10 minutes for review and 10-15 minutes for 
discussion. 


" Facilitate Discussion 

e “What are we potentially missing with this measure that could be 
included to provide for a more valid and useful measure of 
strategic implementation leadership?” 

e “Beyond general leadership qualities, such as caring about 
employee’s well-being or being organized, what specific aspects of 
leadership come to mind when you think about someone who 
would strategically support staff in school to adopt and implement 
evidence-based practices?” 

e What specific things can leaders do that send a message that 
implementation is a high priority in their school? 

e What steps can leaders take to support their staff in successful 
implementation? 

e “What is it about a leader that allows her to promote a positive 
climate for taking on and implementing new practices effectively?” 


Implementation Citizenship Behavior 


e Present the Implementation Citizenship Behavior construct and key elements of the 
construct (provide a handout that outlines the construct and bullet points key idea 
associated with the construct) (2 minutes) 
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“Now, we are going to discuss implementation citizenship behaviors, which are 
measured by the Implementation Citizenship Behavior Scale. Implementation 


citizenship is the extent to which employees in an organization go ‘above and 
beyond’ to support implementation.” 

“Implementation citizenship behaviors are those that demonstrate a commitment 
to EBP by keeping informed about the EBP being implemented and supporting 
colleagues to meet EBP standards. It includes at least two major components: 
Helping Others and Keeping Informed.” 

Helping Others refers to the extent to which employees assist others to make sure 
they implement EBP, help teach EBP implementation procedures, or help others 
with responsibilities related to EBP (e.g., completing fidelity assessments). 
Keeping Informed refers to whether employees keep up to date on changes in EBP 
policy, follow the latest news or new findings regarding EBP, and keep up with 
agency communications related to EBP. 


ICB ACTIVITY 


Part 1 — Individual reflection on ICB (1-2 minutes): 


=> GENERATE IDEAS: “First, we would like to give you each I minute to think 


privately about the kinds of ideas on a blank piece of paper about what 
information should be included in a measure of ICB. [This might be questions that 
you think would be important to ask or specific behaviors or perceptions that we 
should be sure to capture. Even though the measure is included in your packet, we 
ask that you refrain from reviewing it for the time being].” 


Part 2 — Review of original and adapted ICBS measures (10 minutes): 


=> MEASURE REVIEW: Distribute original and adapted ICBS measure form. Have 


each member identify words that may be confusing, items that seem irrelevant for 
the school context, and additional items that seem relevant to capture under the 
measure 
« “In your folder, you have a document that provides the original items 
from the Implementation Citizenship Behavior Scale in one column, our 
adapted version of the item in another column, and a third column for 
your notes/comments. The form also asks for your ratings of item 
appropriateness to schools. After sifting through all the items, we would 
also like you to reflect on additional items that we should consider 
including that would fall under the Implementation Citizenship Behavior 
Scale. These items may include ideas you generated during the first part of 
this activity.” 
« “We will be collecting these forms at the end of the focus group” 
e Provide 5-10 minutes for review and 10-15 minutes for 
discussion. 
* Facilitate discussion 


e “What are we potentially missing with this measure that could be 
included to provide for a more valid and useful measure of 
implementation citizenship behavior?” 

e “Beyond generally positive employee qualities, such as being 
sociable and hardworking, what specific implementation 
citizenship behaviors come to mind when you think about someone 
who would go ‘above and beyond’ to support EBP 
implementation?” 

e “What is it about an employee in an organization that allows her 
to support her colleagues in implementing EBPs effectively?” 

e [NOTE FOR FACILITATOR: Other dimensions in the literature 
include: Sportsmanship — tolerating problems and inconveniences 
without complaining; Organizational compliance — following 
rules and procedures even when no one is looking; and Individual 
initiative — sharing ideas to improve performance, such as voicing 
behaviors, giving extra effort, taking on additional responsibilities] 
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